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Abstract 


This document presents the author's effort to understand the delays involved in publishing an 
idea in the IETF or through the Independent Stream, from the first individual draft to the 
publication of the RFC. We analyze a set of randomly chosen RFCs approved in 2018, looking for 
history and delays. We also use two randomly chosen sets of RFCs published in 2008 and 1998 for 
comparing delays seen in 2018 to those observed 10 or 20 years ago. The average RFC in the 2018 
sample was produced in 3 years and 4 months, of which 2 years and 10 months were spent in the 
working group, 3 to 4 months for IETF consensus and IESG review, and 3 to 4 months in RFC 
production. The main variation in RFC production delays comes from the AUTH48 phase. 


We also measure the number of citations of the chosen RFC using Semantic Scholar, and 
compare citation counts with what we know about deployment. We show that citation counts 
indicate academic interest, but correlate only loosely with deployment or usage of the 
specifications. Counting web references could complement that. 


Status of This Memo 


This document is not an Internet Standards Track specification; it is published for informational 
purposes. 


This is a contribution to the RFC Series, independently of any other RFC stream. The RFC Editor 
has chosen to publish this document at its discretion and makes no statement about its value for 
implementation or deployment. Documents approved for publication by the RFC Editor are not 
candidates for any level of Internet Standard; see Section 2 of RFC 7841. 


Information about the current status of this document, any errata, and how to provide feedback 
on it may be obtained at https://www.rfc-editor.org/info/rfc8963. 
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Copyright Notice 


Copyright (c) 2021 IETF Trust and the persons identified as the document authors. All rights 
reserved. 


This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF 
Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this 
document. Please review these documents carefully, as they describe your rights and restrictions 
with respect to this document. 
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1. Introduction 


As stated on the organization's web site, "The IETF is a large open international community of 
network designers, operators, vendors, and researchers concerned with the evolution of the 
Internet architecture and the smooth operation of the Internet." The specifications produced by 
the IETF are published in the RFC series, along with documents from the IAB, IRTF, and 
Independent streams (as per RFC 8729). In this memo, the author attempts to understand the 
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delays involved in publishing an idea in the IETF or through the Independent Stream, from the 
first individual draft to the publication of the RFC. This is an individual effort, and the author's 
conclusions presented here are personal. There was no attempt to seek IETF consensus. 


The IETF keeps records of documents and process actions in the IETF Datatracker [TRKR]. The 
IETF Datatracker provides information about RFCs and drafts, from which we can infer statistics 
about the production system. We can measure how long it takes to drive a proposition from 
initial draft to final publication, and how these delays can be split between working group 
discussions, IETF reviews, IESG assessment, RFC Editor delays and final reviews by the authors -- 
or, for Independent Stream RFCs, draft production, reviews by the Independent Submissions 
Editor, conflict reviews, RFC Editor delays and final reviews. Tracker data is available for all 
RFCs, not just IETF Stream RFCs. 


Just measuring production delays may be misleading. If the IETF or the other streams simply 
rubber-stamped draft proposals and published them, the delays would be short but the quality 
and impact might suffer. We hope that most of the RFCs that are published are useful, but we 
need a way to measure that usefulness. We try to do that by measuring the number of references 
of the published RFCs in Semantic Scholar [SSCH], and also by asking the authors of each RFC in 
the sample whether the protocols and technologies defined in the RFCs were implemented and 
used on the Internet. The citations measured by the Semantic Scholar include citations in other 
RFCs and in Internet-Drafts. We also measure the number of references on the web, which 
provides some results but would be hard to automate. 


In order to limit the resources required for this study, we selected at random 20 RFCs published 
in 2018, as explained in Section 2.2. The statistical sampling picked both IETF Stream and 
Independent Stream documents. For comparison purposes, we also selected at random 20 RFCs 
published in 1998 and 20 published in 2008. Limiting the sample to 20 out of 209 RFCs published 
in 2018 allows for in-depth analysis of each RFC, but readers should be reminded that the this is 
a small sample. The sample is too small to apply general statistical techniques and quantify 
specific ratios, and discussions of correlation techniques would be inappropriate. Instead, the 
purpose is to identify trends, spot issues, and document future work. 


The information gathered for every RFC in the sample is presented in Section 3. In Section 4, we 
analyze the production process and the sources of delays, comparing the 2018 sample to the 
selected samples for 1998 and 2018. In Section 5.1, we present citation counts for the RFCs in the 
samples, and analyze whether citation counts could be used to evaluate the quality of RFCs. 


The measurement of delays could be automated by processing dates and events recorded in the 
Datatracker. The measurement of published RFCs could be complemented by statistics on 
abandoned drafts, which would measure the efficiency of the IETF triaging process. More 
instrumentation would help understanding how large delays happen during working group 
processes. These potential next steps are developed in Section 6. 
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2. Methodology 


The study reported here started with a simple idea: take a sample of RFCs, and perform an in- 
depth analysis of the path from the first presentation of the idea to its publication, while also 
trying to access the success of the resulting specification. This requires defining the key 
milestones that we want to track, and drawing a random sample using an unbiased process. 


2.1. Defining the Important Milestones 


The IETF Datatracker records a list of events for each document processed by IETF working 
groups. This has a high granularity, and also a high variability. Most documents start life as an 
individual draft, are adopted by a working group, undergo a Working Group Last Call, are 
submitted to the IESG, undergo an JETF Last Call and an IESG review, get eventually approved by 
the IESG, and are processed for publication by the RFC Editor, but there are exceptions. Some 
documents are first submitted to one working group and then moved to another. Some 
documents are published through the Independent Stream, and are submitted to the 
Independent Submissions Editor instead of the IESG. 


In order to simplify tabulation, we break the period from the submission of the first draft to the 
publication of the RFC into three big components: 


e The working group processing time, from the first draft to the start of the IETF last call; 


e The IETF processing time, which lasts from the beginning of the IETF last call to the approval 
by the IESG, including the reviews by various directorates; 


e The RFC production, from approval by the IESG to publication, including the AUTH48 
reviews. 


For submissions to the Independent Stream, we don't have a working group. We consider instead 
the progression of the individual draft until the adoption by the Independent Submissions Editor 
(ISE) as the equivalent of the "Working Group" period, and the delay from adoption by the ISE 
until submission to the RFC Editor as the equivalent of the IETF processing time. 


We measure the starting point of the process using the date of submission of the first draft listed 
on that RFC page in the IETF Datatracker. In most cases, this first draft is an individual draft that 
then resubmitted as a working group draft, or maybe resubmitted with a new name as the draft 
was searching for a home in an IETF working group, or before deciding for submission on the 
Independent Stream. 


The IETF Datatracker entries for RFCs and drafts do not always list working group events like 
Working Group Last Call. The only intermediate event that we list between the first draft and the 
submission to the IESG is the working group adoption, for which we use the date of submission 
of version 00 of the draft eventually published as RFC. We also use that date (of submission of 
version 00) for drafts submitted to the Independent Stream. 
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2.2. Selecting a Random Sample of RFCs 


Basic production mechanisms could be evaluated by processing data from the IETF Datatracker, 
but subjective data requires manual assessment of results, which can be time-consuming. Since 
our resources are limited, we will only perform this analysis for a small sample of RFCs, selected 
at random from the list of RFCs approved in 2018. Specifically, we will pick 20 RFC numbers at 
random between: 


e RFC 8307, published in January 2018, and 
e RFC 8511, published December 2018. 


The list of 20 selected RFCs is: RFC 8411, RFC 8456, RFC 8446, RFC 8355, RFC 8441, RFC 8324, RFC 
8377, RFC 8498, RFC 8479, RFC 8453, RFC 8429, RFC 8312, RFC 8492 , RFC 8378, RFC 8361, RFC 
8472, RFC 8471, RFC 8466, RFC 8362, and RFC 8468. 


When evaluating delays and impact, we will compare the year 2018 to 2008 and 1998, 10 and 20 
years ago. To drive this comparison, we pick 20 RFCs at random among those published in 2008, 
and another 20 among those published in 1998. 


The list of the 20 randomly selected RFCs from 2008 is: RFC 5227, RFC 5174, RFC 5172, RFC 5354, 
RFC 5195, RFC 5236, RFC 5348, RFC 5281, RFC 5186, RFC 5326, RFC 5277, RFC 5373, RFC 5404, RFC 
5329, RFC 5283, RFC 5358, RFC 5142, RFC 5271, RFC 5349, and RFC 5301. 


The list of the 20 randomly selected RFCs from 1998 is: RFC 2431, RFC 2381, RFC 2387, RFC 2348, 
RFC 2391, RFC 2267, RFC 2312, RFC 2448, RFC 2374, RFC 2398, RFC 2283, RFC 2382, RFC 2289, RFC 
2282, RFC 2404, RFC 2449, RFC 2317, RFC 2394, RFC 2297, and RFC 2323. 


2.3. Conventions Used in This Document 


The following abbreviations are used in the tables: 


BCP Best Current Practice 

Exp Experimental 

Info Informational 

PS Proposed Standard 

DS Draft Standard [This maturity level was retired by RFC 6410.] 


In addition, Status is as defined in RFC 2026, and Stream is as defined in RFC 8729. 


3. Analysis of 20 Selected RFCs 


We review each of the RFCs listed in Section 2.2 for the year 2018, trying both to answer the 
known questions and to gather insight for further analyses. In many cases, the analysis of the 
data is complemented by direct feedback from the RFC authors. 
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3.1. RFC 8411 
"IANA Registration for the Cryptographic Algorithm Object Identifier Range" [RFC8411]: 


Status (Length): Informational (5 pages) 
Overview: 4 individual drafts 
First draft: 2017-05-08 

Last Call start: 2017-10-09 

IESG eval. start: 2017-12-28 

IESG approved: 2018-02-26 (draft 03) 
AUTH48 start: 2018-04-20 

AUTH48 complete: 2018-07-17 

Published: 2018-08-06 

IANA action: create table 


This RFC was published from the individual draft, which was not resubmitted as a working 
group draft. 


The draft underwent minor copy editing before publication. 


Some but not all of the long delay in AUTH48 is due to clustering with [RFC8410]. MISSREF state 
concluded on 2018-05-09 and the document re-entered AUTH48 at once. AUTH48 lasted over two 
months after that. (For state definitions, see <https://www.rfc-editor.org/about/queue/ 
#state_def>.) 


The time after AUTH48 and before publication (3 weeks) partly overlaps with travel for IETF 102 
and is partly due to coordinating the cluster. 


3.2. RFC 8456 


"Benchmarking Methodology for Software-Defined Networking (SDN) Controller Performance" 
[RFC8456]: 


Status (Length): Informational (64 pages) 
Overview: 2 individual drafts; 9 WG drafts 
First draft: 2015-03-23 

WG adoption: 2015-10-18 

Last Call start: 2018-01-19 

IESG eval. start: 2018-02-27 

IESG approved: 2018-05-25 

AUTH48 start: 2018-08-31 

AUTH48 complete: 2018-10-16 

Published: 2018-10-30 
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The draft underwent extensive copy editing, covering use of articles, syntax, and word choice. 
The changes are enough to cause pagination differences. The "diff" tool marks pretty much every 
page as changed. Some diagrams see change in protocol elements like message names. 


According to the author, the experience of producing this document mirrors a typical one in the 
Benchmarking Methodologies Working Group (BMWG). There were multiple authors in multiple 
time zones, which slowed down the AUTH48 process somewhat, although the AUTH48 delay of 46 
days is only a bit longer than the average draft. 


The RFC was part of cluster with [RFC8455]. 


BMWG publishes Informational RFCs centered around benchmarking, and the methodologies in 
RFC 8456 have been implemented in benchmarking products. 


3.3. RFC 8446 


"The Transport Layer Security (TLS) Protocol Version 1.3" [RFC8446], as the title indicates, defines 
the new version of the TLS protocol. From the IETF Datatracker, we extract the following: 


Status (Length): Proposed Standard (160 pages) 
Overview: 29 WG drafts 

First draft: 2014-04-17 

Last Call start: 2018-02-15 

IESG eval. start: 2018-03-02 

IESG approved: 2018-03-21 (draft 28) 

AUTH48 start: 2018-06-14 

AUTH48 complete: 2018-08-10 

Published: 2018-08-10 


This draft started as a WG effort. 


The RFC was a major effort in the IETF. Working group participants developed and tested several 
implementations. Researchers analyzed the specifications and performed formal verifications. 
Deployment tests outlined issues that caused extra work when the specification was almost 
ready. This complexity largely explains the time spent in the working group. 


Comparing the final draft to the published version, we find relatively light copy editing. It 
includes explaining acronyms on first use, clarifying some definitions standardizing punctuation 
and capitalization, and spelling out some numbers in text. This generally fall in the category of 
"style", although some of the clarifications go into message definitions. However, that simple 
analysis does not explain why the AUTH48 phase took almost two months. 


This document's AUTH48 process was part of the "GitHub experiment", which tried to use GitHub 
pull requests to track the AUTH48 changes and review comments. The RFC Production Center 
(RPC) staff had to learn using GitHub for that process, and this required more work than the 
usual RFC. The author and AD thoroughly reviewed each proposed edit, accepting some and 
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rejecting some. The concern there was that any change in a complex specification might affect a 
protocol that was extensively reviewed in the working group, but of course these reviews added 
time to the AUTH48 delays. 


There are 21 implementations listed in the Wiki of the TLS 1.3 project [TLS13IMP]. It has been 
deployed on major browsers, and is already used in a large fraction of TLS connections. 


3.4. RFC 8355 


"Resiliency Use Cases in Source Packet Routing in Networking (SPRING) Networks" [RFC8355] is 
an Informational RFC. It originated from an informational use-case draft; it was mostly used for 
the BOF creating the WG, and then to drive initial work and evolutions from the WG. 


Status (Length): Informational (13 pages) 
Overview: 2 individual drafts; 13 WG drafts 
First draft: 2014-01-31 

WG adoption: 2014-05-13 

Last Call start: 2017-04-20 

IESG eval. start: 2017-05-04 (draft 09) 

IESG approved: 2017-12-19 (draft 12) 

AUTH48 start: 2018-03-12 

AUTH48 complete: 2018-03-27 

Published: 2018-03-28 


Minor set of copy edits, mostly for style. 


No implementation of the RFC itself, but the technology behind it (such as Segment Routing 
Architecture [RFC8402] and TI-LFA [TI-LFA]) is widely implemented and deployment is ongoing. 


According to participants in the discussion, the process of adoption of the source packet routing 
standards was very contentious. The establishment of consensus at both the working group level 
and the IETF level was difficult and time-consuming. 


3.5. RFC 8441 
"Bootstrapping WebSockets with HTTP/2" [RFC8441] 


Status (Length): Proposed Standard (8 pages) 

Overview: 3 individual drafts; 8 WG drafts; Updates RFC 6455 
First draft: 2017-10-15 

WG adoption: 2017-12-19 

Last Call start: 2018-05-07 (draft 05) 

IESG eval. start: 2018-05-29 (draft 06) 

IESG approved: 2018-06-18 (draft 07) 

AUTH4S8 start: 2018-08-13 


AUTH48 complete: 2018-09-15 
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Published: 2018-09-18 
IANA action: table entries 


This RFC defines the support of WebSockets in HTTP/2, which is different from the mechanism 
defined for HTTP/1.1 in [RFC6455]. The process was relatively straightforward, involving the 
usual type of discussions, some on details and some on important points. 


Comparing the final draft and published RFC shows a minor set of copy edits, mostly for style. 
However, the author recalls a painful process. The RFC includes many charts and graphs that 
were very difficult to format correctly in the author's production process that involved 
conversions from markdown to XML, and then from XML to text. The author had to get 
substantial help from the RFC Editor. 


There are several implementations, including Firefox and Chrome, making RFC 8441 a very 
successful specification. 


3.6. RFC 8324 


"DNS Privacy, Authorization, Special Uses, Encoding, Characters, Matching, and Root Structure: 
Time for Another Look?" [RFC8324]. This is an opinion piece on DNS development, published on 
the Independent Stream. 


Status (Length): Informational (29 pages) 
Overview: 5 individual drafts; Independent Stream 
First draft: 2017-06-02 


ISE review start: 2017-07-10 (draft 03) 
IETF conflict review start: 2017-10-29 


Approved: 2017-12-18 (draft 04) 
AUTH48 start: 2018-01-29 (draft 05) 
AUTH48 complete: 2018-02-26 
Published: 2018-02-27 


This RFC took only 9 months from first draft to publication, which is the shortest in the 2018 
sample set. In part, this is because the text was privately circulated and reviewed by the ISE's 
selected experts before the first draft was published. The nature of the document is another 
reason for the short delay. It is an opinion piece and does not require the same type of consensus 
building and reviews as a protocol specification. 


Comparing the final draft and the published version shows only minor copy edits, mostly for 
style. According to the author, this is because he knows how to write in RFC style with the result 
that his documents often need a minimum of editing. He also makes sure that the document on 
which the RFC Production Center starts working already has changes discussed and approved 
during Last Call and IESG review incorporated, rather than expecting the Production Center to 
operate off of notes about changes to be made. 
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3.7. RFC 8377 
"Transparent Interconnection of Lots of Links (TRILL): Multi-Topology" [RFC8377] 


Status (Length): Proposed Standard (20 pages) 
Overview: 3 individual drafts; 7 WG drafts; Updates RFCs 6325 and 7177 
First draft: 2013-09-03 

WG adoption: 2015-09-01 

Last Call start: 2018-02-19 (draft 05) 

IESG eval. start: 2018-03-06 (draft 05) 

IESG approved: 2018-03-12 (draft 06) 

AUTH48 start: 2018-04-20 (draft 06) 

AUTH48 complete: 2018-07-31 

Published: 2018-07-31 

IANA action: table entries 


Minor set of copy edits, mostly for style, also clarity. 


3.8. RFC 8498 


"A P-Served-User Header Field Parameter for an Originating Call Diversion (CDIV) Session Case in 
the Session Initiation Protocol (SIP)" [RFC8498]. 


Status (Length): Informational (15 pages) 
Overview: 5 individual drafts; 9 WG drafts 
First draft: 2016-03-21 

WG adoption: 2017-05-15 

Last Call start: 2018-10-12 (draft 05) 
IESG eval. start: 2018-11-28 (draft 07) 
IESG approved: 2018-12-11 (draft 08) 
AUTH48 start: 2019-01-28 

AUTH48 complete: 2019-02-13 

Published: 2019-02-14 

IANA action: table rows added. 


Copy edits for style, but also clarification of ambiguous sentences. 


3.9. RFC 8479 
"Storing Validation Parameters in PKCS#8" [RFC8479] 


Status (Length): Informational (8 pages) 
Overview: 5 individual drafts; Independent Stream 
First draft: 2017-08-08 


Huitema Informational Page 11 


RFC 8963 RFC Evaluation 2018 January 2021 


ISE review start: 2018-12-10 (draft 00) 
IETF conflict review start: 2018-03-29 


Approved: 2018-08-20 (draft 03) 
AUTH48 start: 2018-09-20 (draft 04) 
AUTH48 complete: 2018-09-25 
Published: 2018-09-26 


The goal of the draft was to document what the gnutls implementation was using for storing 
provably generated RSA keys. This is a short RFC that was published relatively quickly, although 
discussion between the author, the Independent Submissions Editor, and the IESG lasted several 
months. In the initial conflict review, the IESG asked the ISE to not publish this document before 
IETF working groups had an opportunity to pick up the work. The author met that requirement 
by a presentation to the SECDISPATCH WG during IETF 102. Since no WG was interested in 
picking up the work, the document progressed on the Independent Stream. 


Very minor set of copy edits, moving some references from normative to informative. 


The author is not aware of other implementations than gnutls relying on this RFC. 


3.10. RFC 8453 
"Framework for Abstraction and Control of TE Networks (ACTN)" [RFC8453] 


Status (Length): Informational (42 pages) 
Overview: 3 individual drafts; 16 WG drafts 
First draft: 2015-06-15 

WG adoption: 2016-07-15 

Out of WG: 2018-01-26 (draft 11) 
Expert review requested: 2018-02-13 

Last Call start: 2018-04-16 (draft 13) 
IESG eval. start: 2018-05-16 (draft 14) 
IESG approved: 2018-06-01 (draft 15) 
AUTH48 start: 2018-08-13 

AUTH48 complete: 2018-08-20 

Published: 2018-08-23 

IANA action: table rows added. 

Minor copy editing. 


3.11. RFC 8429 
"Deprecate Triple-DES (3DES) and RC4 in Kerberos" [RFC8429] 


Status (Length): BCP (10 pages) 
Overview: 6 WG drafts 
First draft: 2017-05-01 
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Last Call start: 
IESG eval. start: 
IESG approved: 
AUTH48 start: 
AUTH48 complete: 
Published: 

IANA action: 


RFC Evaluation 2018 


2017-07-16 (draft 03) 
2017-08-18 (draft 04) 
2018-05-25 (draft 05) 
2018-07-24 
2018-10-31 
2018-10-31 

table rows added. 


This draft started as a working group effort. 


January 2021 


This RFC recommends deprecating two encryption algorithms that are now considered obsolete 
and possibly broken. The document was sent back to the WG after the first Last Call, edited, and 
then there was a second Last Call. The delay from first draft to Working Group Last Call was 
relatively short, but the number may be misleading. The initial draft was a replacement of a 
similar draft in the KITTEN Working Group, which stagnated for some time before the CURDLE 
Working Group took up the work. The deprecation of RC4 was somewhat contentious, but the 
WG had already debated this prior to the production of this draft, and the draft was not delayed 


by this debate. 


Most of the 280 days between IETF LC and IESG approval were because the IESG had to talk 
about whether this document should obsolete RFC 4757 or move it to Historic status, and no one 
was really actively pushing that discussion for a while. 


The 99 days in AUTH48 are mostly because one of the authors was a sitting AD, and those duties 


ended up taking precedence over reviewing this document. 


Minor copy editing, for style. 


The implementation of the draft would be the actual removal of support for 3DES and RC4 in 


major implementations. This is happening, but very slowly. 


3.12. RFC 8312 


"CUBIC for Fast Long-Distance Networks" [RFC8312] 


Status (Length): 
Overview: 

First draft: 

WG adoption: 
Last Call start: 
IESG eval. start: 
IESG approved: 
AUTH48 start: 
AUTH48 complete: 
Published: 
IANA action: 


Huitema 


Informational (18 pages) 
2 individual drafts; 8 WG drafts 
2014-09-01 

2015-06-08 

2017-09-18 (draft 06) 
2017-10-04 

2017-11-14 (draft 07) 
2018-01-08 

2018-02-07 

2018-02-07 

table rows added. 


Informational 
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Minor copy editing, for style. 


The TCP congestion control algorithm Cubic was first defined in 2005, was implemented in Linux 
soon after, and was implemented in major OSes after that. After some debates from 2015 to 2015, 
the TCPM Working Group adopted the draft, with a goal of documenting Cubic in the RFC Series. 
According to the authors, this was not a high-priority effort, as Cubic was already implemented 
in multiple OSes and documented in research papers. At some point, only one of the authors was 
actively working on the draft. This may explain why another two years was spent progressing 
the draft after adoption by the WG. 


The RFC publication may or may not have triggered further implementations. On the other hand, 
several OSes picked up bug fixes from the draft and the RFC. 


3.13. RFC 8492 
"Secure Password Ciphersuites for Transport Layer Security (TLS)" [RFC8492] 


Status (Length): Informational (40 pages) 

Overview: 10 individual drafts; 8 WG drafts; Independent Stream 
First draft: 2012-09-07 

Targeted to ISE: 2016-08-05 


ISE review start: 2017-05-10 (draft 01) 
IETF conflict review start: 2017-09-04 


Approved: 2017-10-29 (draft 02) 
AUTH48 start: 2018-10-19 (draft 05) 
AUTH48 complete: 2019-02-19 
Published: 2019-02-21 

IANA action: table rows added. 


This RFC has a complex history. The first individual draft was submitted to the TLS Working 
Group on September 7, 2012. It progressed there, and was adopted by the WG after 3 revisions. 
There were then 8 revisions in the TLS WG, until the WG decided to not progress it. The draft was 
parked in 2013 by the WG chairs after failing to get consensus in WG Last Call. The AD finally 
pulled the plug in 2016, and the draft was then resubmitted to the ISE. 


At that point, the author was busy and was treating this RFC with a low priority because, in his 
words, it would not be a "real RFC". There were problems with the draft that only came up late. 
In particular, it had to wait for a change in registry policy that only came about with the 
publication of TLS 1.3, which caused the draft to be published after RFC 8446, and also required 
adding references to TLS 1.3. The author also got a very late comment while in AUTH48 that 
caused some rewriting. Finally, there was some IANA issue with the extension registry where a 
similar extension was added by someone else. The draft was changed to just use it. 


Changes in AUTH48 include adding a reference to TLS 1.3, copy editing for style, some added 
requirements, added paragraphs, and changes in algorithms specification. 
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3.14. RFC 8378 


"Signal-Free Locator/ID Separation Protocol (LISP) Multicast" [RFC8378] is an Experimental RFC, 
defining how to implement Multicast in the LISP architecture. 


Status (Length): Experimental (21 pages) 
Overview: 5 individual drafts; 10 WG drafts 
First draft: 2014-02-28 

WG adoption: 2015-12-21 

Last Call start: 2018-02-13 (draft 07) 

IESG eval. start: 2018-02-28 (draft 08) 

IESG approved: 2018-03-12 (draft 09) 

AUTH48 start: 2018-04-23 

AUTH48 complete: 2018-05-02 

Published: 2018-05-02 


Preparing the RFC took more than 4 years. According to the authors, they were not aggressively 
pushing it and just let the working group process decide to pace it. They also did 
implementations during that time. 


Minor copy editing, for style. 


The RFC was implemented by lispers.net and Cisco, and it was used in doing IPv6 multicast over 
IPv4 unicast/multicast at the Olympics in PyeungChang. The plan is to work on a Proposed 
Standard once the experiment concludes. 


3.15. RFC 8361 


"Transparent Interconnection of Lots of Links (TRILL): Centralized Replication for Active-Active 
Broadcast, Unknown Unicast, and Multicast (BUM) Traffic" [RFC8361] 


Status (Length): Proposed Standard (17 pages) 
Overview: 3 individual drafts; 14 WG drafts 
First draft: 2013-11-12 

WG adoption: 2014-12-16 

Last Call start: 2017-11-28 (draft 10) 

IESG eval. start: 2017-12-18 (draft 11) 

IESG approved: 2018-01-29 (draft 13) 

AUTH48 start: 2018-03-09 

AUTH48 complete: 2018-04-09 

Published: 2018-04-12 


According to the authors, the long delays in producing this RFC were due to a slow uptake of the 
technology in the industry. 
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Minor copy editing, for style. 


There was at least one partial implementation. 


3.16. RFC 8472 
"Transport Layer Security (TLS) Extension for Token Binding Protocol Negotiation" [RFC8472] 


Status (Length): Proposed Standard (8 pages) 
Overview: 1 individual draft; 15 WG drafts 
First draft: 2015-05-29 

WG adoption: 2015-09-11 

Last Call start: 2017-11-13 (draft 10) 

IESG eval. start: 2018-03-19 

IESG approved: 2018-07-20 (draft 14) 

AUTH48 start: 2018-09-17 

AUTH48 complete: 2018-09-25 

Published: 2018-10-08 


This is a pretty simple document, but it took over 3 years from individual draft to RFC. According 
to the authors,the biggest setbacks occurred at the start: it took a while to find a home for this 
draft. It was presented in the TLS WG (because it's a TLS extension) and UTA WG (because it has 
to do with applications using TLS). Then the ADs determined that a new WG was needed, so the 
authors had to work through the WG creation process, including running a BOF. 


Minor copy editing, for style, with the addition of a reference to TLS 1.3. 


Perhaps partially due to the delays, some of the implementers lost interest in supporting this 
RFC, 


3.17. RFC 8471 
"The Token Binding Protocol Version 1.0" [RFC8471] 


Status (Length): Proposed Standard (18 pages) 
Overview: 1 individual draft; 19 WG drafts 
First draft: 2014-10-13 

WG adoption: 2015-03-15 

Last Call start: 2017-11-13 (draft 16) 

IESG eval. start: 2018-03-19 

IESG approved: 2018-07-20 (draft 19) 

AUTH48 start: 2018-09-17 

AUTH48 complete: 2018-09-25 

Published: 2018-10-08 
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This document presents a Token Binding Protocol for TLS. We can notice a period of 5 months 
before adoption of the draft by the WG. That explains in part the overall time of almost 4 years 
from first draft to publication. 


Minor copy editing, for style. 


The web references indicate adoption in multiple development projects. 


3.18. RFC 8466 
"A YANG Data Model for Layer 2 Virtual Private Network (L2VPN) Service Delivery" [RFC8466] 


Status (Length): Proposed Standard (158 pages) 
Overview: 5 individual drafts; 11 WG drafts 
First draft: 2016-09-01 

WG adoption: 2017-02-26 

Last Call start: 2018-02-21 (draft 07) 

IESG eval. start: 2018-03-14 (draft 08) 

IESG approved: 2018-06-25 (draft 10) 

AUTH48 start: 2018-09-17 

AUTH48 complete: 2018-10-09 

Published: 2018-10-12 


Copy editing for style and clarity, with also corrections to the YANG model. 


3.19. RFC 8362 


"OSPFv3 Link State Advertisement (LSA) Extensibility" [RFC8362] is a major extension to the 
OSPF protocol. It makes OSPFv3 fully extensible. 


Status (Length): Proposed Standard (33 pages) 
Overview: 4 individual drafts; 24 WG drafts 
First draft: 2013-02-17 

WG adoption: 2013-10-15 

Last Call start: 2017-12-19 (draft 19) 

IESG eval. start: 2018-01-18 (draft 20) 

IESG approved: 2018-01-29 (draft 23) 

AUTH48 start: 2018-03-19 

AUTH48 complete: 2018-03-30 

Published: 2018-04-03 


The specification was first submitted as an individual draft in the IPv6 WG, then moved to the 
OSPF WG. The long delay of producing this RFC is due to the complexity of the problem, and the 
need to wait for implementations. It is a very important change to OSPF that makes OSPFv3 fully 
extensible. Since it was a non-backward compatible change, the developers started out with 
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some very complex migration scenarios but ended up with either legacy or extended OSPFv3 
LSAs within an OSPFv3 routing domain. The initial attempts to have a hybrid mode of operation 
with both legacy and extended LSAs also delayed implementation due to the complexity. 


Copy editing for style and clarity. 


This specification either was or will be implemented by all the router vendors. 


3.20. RFC 8468 


"IPv4, IPv6, and IPv4-IPv6 Coexistence: Updates for the IP Performance Metrics (IPPM) 
Framework" [RFC8468]. 


Status (Length): Informational (15 pages) 
Overview: 3 individual drafts; 7 WG drafts 
First draft: 2015-08-06 

WG adoption: 2016-07-04 

Last Call start: 2018-04-11 (draft 04) 

IESG eval. start: 2018-05-24 (draft 05) 

IESG approved: 2018-07-10 (draft 06) 

AUTH48 start: 2018-09-13 

AUTH48 complete: 2018-11-05 

Published: 2018-11-14 


RFC 8468 was somehow special in that there was not a technical reason or interest that triggered 
it, but rather a formal requirement. While writing RFC 7312, the IP Performance Metrics (IPPM) 
Working Group realized that RFC 2330, the IP Performance Metrics Framework supported IPv4 
only and explicitly excluded support for IPv6. Nevertheless, people used the metrics that were 
defined on top of RFC 2330 (and, therefore, IPv4 only) for IPv6, too. Although the IPPM WG 
agreed that the work was needed, the interest of IPPM attendees in progressing (and reading/ 
reviewing) the IPv6 draft was limited. Resolving the IPv6 technical part was straightforward, but 
subsequently some people asked for a broader scope (topics like header compression, 6LOWPAN, 
etc.), and it took some time to figure out and later on convince people that these topics are out of 
scope. The group also had to resolve contentious topics, for example, how to measure the 
processing of IPv6 extension headers, which is sometimes nonstandard. 


The time in AUTH48 state for this document was longer than average. According to the authors, 
the main reasons include: 


e Workload and travel caused by busy work periods of all coauthors 


e Time zone difference between coauthors and editor (at least US, Europe, and India, not 
considering travel) 


e RFC Production Center proposed and committed some unacceptable modifications that 
needed to be reverted 


Huitema Informational Page 18 


RFC 8963 RFC Evaluation 2018 January 2021 


e Lengthy discussions on a new document title (required high effort and took a long time, in 
particular reaching consensus between coauthors and editor was time-consuming and 
involved the AD) 

e RFC Production Center correctly identified some nits (obsoleted personal websites of 
coauthors) and coauthors attempting to fix them. 


The differences between the final draft and the published RFC show copy editing for style and 
clarity, but do not account for the back and forth between authors and editors mentioned by the 
authors. 


4. Analysis of Process and Delays 


We examine the 20 RFCs in the sample, measuring various characteristics such as delay and 
citation counts, in an attempt to identify patterns in the IETF processes. 


4.1. Delays from First Draft to RFC 


We look at the distribution of delays between the submission of the first draft and the 
publication of the RFC, using the three milestones defined in Section 2.1: processing time in the 
working group, IETF processing time, and RFC production time. The following table shows the 
number of days in each phase for the 20 RFCs in the sample: 


RFC Status Pages Overall WG IETF Edit 


8411 Info 5 455 154 140 161 
8456 Info 64 1317 1033 126 158 
8446 PS 160 1576 1400 34 142 
8355 Info 13 1517 1175 243 99 
8441 PS 8 32 204 31 92 
8324 Info (ISE) 29 270 38 161 71 
8377 PS 8 1792 1630 2 141 
8498 Info 15 1059 935 59 65 
8479 Info (ISE) 8 414 233 144 37 
8453 Info 42 1165 1036 46 83 
8429 BCP 10 548 76 313 159 
8312 Info 18 1214 1113 16 85 
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RFC Status Pages Overall WG IETF Edit 


8492 Info (ISE) 40 2358 1706 172 480 
8378 Exp 21 1524 1446 27 51 
8361 PS 17 1612 1477 62 73 
8472 PS 8 1228 899 249 80 
8471 PS 18 1228 899 249 80 
8466 PS 158 771 538 124 109 
8362 PS 33 1871 1766 41 64 
8468 Info 15 1196 979 90 127 
average 35 1172 948 ily ie 
average (not ISE) 36 1200 999 110 104 
Table 1 


The average delay from first draft to publication is about 3 years and 3 months, but this varies 
widely. Excluding the RFCs from the Independent Stream, the average delay from start to finish is 
3 years and 4 months, of which on average 2 years and 9 months are spent getting consensus in 
the working group, and 3 to 4 months each for IETF consensus and for RFC production. 


The longest delay is found for [RFC8492], 6.5 years from start to finish. This is however a very 
special case -- a draft that was prepared for the TLS Working Group and failed to reach 
consensus. After that, it was resubmitted to the ISE, and incurred atypical production delays. 


On average, we see that 80% of the delay is incurred in WG processing, 10% in IETF review, and 
10% for edition and publication. 


For IETF Stream RFCs, it appears that the delays for Informational documents are slightly shorter 
than those for protocol specifications, maybe six months shorter on average. However, there are 
lots of differences between individual documents. The delays range from less than a year to more 
than 5 years for protocol specifications, and from a year and 3 months to a bit more than 4 years 
for Informational documents. 


We can compare the delays in the 2018 samples to those observed 10 years ago and 20 years 
before: 


RFC (2008) Status Pages Delay 


5326 Exp 54 1584 
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RFC (2008) Status Pages 


5348 PS 58 
5281 Info 51 
5354 Exp 23 
5227 PS 21 
5329 PS 12 
S27 PS 35 
5236 Info (ISE) 26 
5358 BCP 7 
5271 Info 22 
5195 PS 10 
5283 PS 12 
5186 Info 6 
5142 PS 13 
5373 PS 24 
5404 PS 27 
510172 PS 7 
5349 Info 10 
5301 PS 6 
5174 Info 8 
Table 2 


RFC (1998) Status Pages 


2289 PS 25 

2267 Info 10 

Zane BCP 10 
Informational 


Delay 
823 
1308 
2315 
2434 
1980 
912 
1947 
884 
1066 
974 
1096 
2253 
1005 
1249 
214 
305 
1096 
396 


427 


Delay 
396 
unknown 


485 
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RFC (1998) Status Pages 


2404 


2374 


2449 


2283 


2394 


2348 


2382 


2297 


2381 


2312 


2387 


2398 


2391 


2431 


2282 


2929 


2448 
Table 3 


PS 

PS 

PS 

PS 

Info 

DS 

Info 

Info (ISE) 
PS 

Info 

PS 

Info 

PS 

PS 

Info 

Info (ISE) 


Info (ISE) 


7 


12 


19 


5 


Delay 
488 
289 
273 
153 
365 
699 
396 

28 

699 

365 

122 

396 

122 

457 

215 
unknown 


92 


January 2021 


We can compare the median delay, and the delays observed by the fastest and slowest quartiles 


in the three years: 


Huitema 


Year 


2018 


2008 


1998 
Table 4 


Fastest 25% Median 


TUS) 122 
869 108 
169 36 

Informational 


1 


1 


5 


Slowest 25% 


1537 


1675 


442 
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The IETF takes three to four times more to produce an RFC in 2018 than it did in 1998, but about 
the same time as it did in 2008. We can get a rough estimate of how this translates in terms of 
"level of attention" per RFC by comparing the number of participants in the IETF meetings of 
2018, 2008, and 1998 [IETFCOUNT] to the number of RFCs published these years [RFCYEAR]. 


Year Number of Spring Summer Fall Average Attendees/ 
RFCs E: P. P. P. RFC 
2018 208 1235 1078 879 1064 5.1 
2008 290 1128 1181 962 1090 3.8 
1998 234 1775 2106 1705 1862 8.0 
Table 5 


The last column in the table provides the ratio of average number of participants to the number 
of RFCs published. If the IETF were a centralized organization, and if all participants and 
documents were equivalent, this ratio would be the number of participants dedicated to produce 
an RFC on a given year. This is of course a completely abstract figure because none of the 
hypotheses above are true, but it still gives a vague indication of the "level of attention" applied 
to documents. We see that this ratio has increased from 2008 to 2018, as the number of 
participants was about the same for these two years but the number of published RFCs 
decreased. However, this ratio was much higher in 1998. The IETF had many more participants, 
and there were probably many more eyes available to review any given draft. If we applied the 
ratios of 1998, the IETF would be producing 119 documents in 2018 instead of 208. 


4.2. Working Group Processing Time 


The largest part of the delays is spent in the working groups, before the draft is submitted to the 
IESG for IETF review. As mentioned in Section 2.1, the only intermediate milestone that we can 
extract from the IETF Datatracker is the date at which the document was adopted by the working 
group, or targeted for independent submission. The breakdown of the delays for the documents 
in our sample is: 


RFC Status WG Until adoption After adoption 
8411 Info 154 0 154 
8456 Info 1033 209 824 
8446 PS 1400 0 1400 
8355 Info 1175 102 1073 
8441 PS 204 65 139 
8324 Info (ISE) 38 0 38 
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RFC Status 
8377 PS 
8498 Info 
8479 Info (ISE) 
8453 Info 
8429 BCP 
8312 Info 
8492 Info (ISE) 
8378 Exp 
8361 PS 
8472 PS 
8471 PS 
8466 PS 
8362 PS 
8468 Info 
Average 
Table 6 


RFC Evaluation 2018 


WG Until adoption 


1630 


935 


233 


1036 


76 


1113 


1706 


1446 


1477 


899 


T2 


538 


1766 


979 


948 


728 


399 


178 


240 


333 


285 


After adoption 
902 
515 
233 
640 

76 
833 
278 
785 

1078 
794 
794 
360 
1526 
646 


663 


January 2021 


The time before working group adoption averages to a bit more than 9 months, compared to 1 
year and almost 10 months for processing time after adoption. We see that RFC 8492 stands out, 
with long delays spent attempting publication through a working group before submission to the 
Independent Submissions Editor. If we remove RFC 8492 from the list, the average time until 
adoption drops to just over 7 months, and becomes just 25% of the total processing time in the 


WG. 


There are a few documents that started immediately as working group efforts, or were 


immediately targeted for publication in the Independent Stream. Those documents tend to see 
short processing times, with the exception of RFC 8446 on which the TLS Working Group spent a 
long time working. 
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4.3. Preparation and Publication Delays 


The preparation and publication delays include three components: 


January 2021 


e the delay from submission to the RFC Editor to beginning of AUTH48, during which the 


document is prepared (referred to as "RFC edit" below); 


e the AUTH48 delay, during which authors review and eventually approve the changes 
proposed by the editors (referred to as "AUTH48" below); 


e the publication delay, from final agreement by authors and editors to actual publication 
(referred to as "RFC Pub" below). 


The breakdown of the publication delays for each RFC is shown in the following table. 


Huitema 


RFC 


8411 


8456 


8446 


8355 


8441 


8324 


8377 


8498 


8479 


8453 


8429 


8312 


8492 


8378 


8361 


8472 


8471 


Status 
Info 

Info 

PS 

Info 

PS 

Info (ISE) 
PS 

Info 

Info (ISE) 
Info 

BCP 

Info 

Info (ISE) 
Exp 

PS 

PS 


PS 


Pages 


5 


17 


18 


RFC edit AUTH48 


Be 


98 


85 


83 


56 


42 


39 


48 


al 


73 


60 


55 


359 


42 


39 


59 


59 


Informational 


88 


99 


28 


RFC Pub Edit (total) 


20 


14 


13 


13 


161 
158 
142 
99 
97; 
71 
141 
65 
37 
83 
159 
85 
480 
51 
1 
80 


80 
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RFC Status Pages RFCedit AUTH48 
8466 PS 158 84 22 
8362 PS 33 49 T 
8468 Info 15 65 53 
Average 74 39 
Average (without 8492) 59 35 
Table 7 


RFC Pub 


3 


January 2021 


Edit (total) 
109 


64 


99 


On average, the total delay appears to be about four months, but the average is skewed by the 
extreme values encountered for [RFC8492]. If we exclude that RFC from the computations, the 
average delay drops to a just a bit more than 3 months: about 2 months for the preparation, a bit 
more than one month for the AUTH48 phase, and 5 days for the publishing. 


Of course, these delays vary from RFC to RFC. To try explain the causes of the delay, we compute 
the correlation factor between the observed delays and several plausible explanation factors: 


e the number of pages in the document, 


e the amount of copy editing, as discussed in Section 4.4, 


e whether or not IANA actions were required, 


e the number of authors, 


e the number of draft revisions, 


e the working group delay. 


We find the following values: 


Huitema 


Correlation RFC edit AUTH48 
Number of pages 0.50 -0.04 
Copy-Edit 0.42 0.24 
IANA -0.14 -0.21 
Number of authors 0.39 -0.07 
Number of drafts 0.18 -0.33 
WG delay 0.03 -0.16 
Table 8 
Informational 


Edit(total) 
0.21 

0.45 

0.12 

0.18 

-0.19 


-0.15 
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We see some plausible explanations for the production delay. It will be somewhat longer for 
longer documents or for documents that require a lot of copy editing (see Section 4.4). Somewhat 
surprisingly, it also tends to increase with the number of authors. It does not appear significantly 
correlated with the presence or absence of IANA action. 


The analysis of RFC 8324 in Section 3.6 explains its short editing delays by the experience of the 
author. This makes sense: if a document needs less editing, the editing delays would be shorter. 
This is partially confirmed by the relation between the amount of copy editing and the 
publication delay. 


We see fewer plausible explanations for the AUTH48 delays. These delays vary much more than 
the preparation delay, with a standard deviation of 20 days for AUTH48 versus 10 days for the 
preparation delay. In theory, AUTH48 is just a final verification: the authors receive the 
document prepared by the RFC production center, and just have to give their approval, or maybe 
request a last minute correction. The name indicates that this is expected to last just two days, 
but in average it lasts more than a month. 


We often hypothesize that the number of authors influences the AUTH48 delay, or that authors 
who have spent a long time working on the document in the working group somehow get 
demotivated and spend even longer to answer questions during AUTH48. This may happen 
sometimes, but our statistics don't show that - if anything, the numerical results point in the 
opposite direction. 


After asking the authors of the RFCs in the sample why the AUTH48 phase took a long time, we 
got three explanations: 


1. Some RFCs have multiple authors in multiple time zones. This slows down the coordination 
required for approving changes. 


2. Some authors found some of the proposed changes unnecessary or undesirable, and asked 
that they be reversed. This required long exchanges between authors and editors. 


3. Some authors were not giving high priority to AUTH48 responses. 


As mentioned above, we were not able to verify these hypotheses by looking at the data. The 
author's experience with this document suggests another potential delay for the Independent 
Stream RFC: processing delay by the Independent Submissions Editor, discussed in Section 4.5. 


4.4. Copy Editing 


We can assess the amount of copy editing applied to each published RFC by comparing the text of 
the draft approved for publication and the text of the RFC. We do expect differences in the 
"boilerplate" and in the IANA section, but we will also see differences due to copy editing. 
Assessing the amount of copy editing is subjective, and we do it using a scale of 1 to 4: 


1: Minor editing 


2: Editing for style, such as capitalization, hyphens, "that" versus "which", and expanding all 
acronyms at least once. 
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Editing for clarity in addition to style, such as rewriting ambiguous sentences and clarifying 
use of internal references. For YANG models, that may include model corrections suggested 


by the verifier. 


Extensive editing. 


The following table shows that about half of the RFCs required editing for style, and the other 
half at least some editing for clarity. 
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RFC Status Copy Edit 
8468 Info 3 
Table 9 


This method of assessment does not take into account the number of changes proposed by the 
editors and eventually rejected by the authors, since these changes are not present in either the 
final draft or the published RFC. It might be possible to get an evaluation of these "phantom 
changes" from the RFC Production Center. 


4.5. Independent Stream 


Out of 20 randomly selected RFCs, 3 were published through the Independent Stream. One is an 
independent opinion, another a description of a non-IETF protocol format, and the third was 
[RFC8492], which is a special case. Apart from this special case, the publication delays were 
significantly shorter for the Independent Stream than for the IETF Stream. 


The authors of these 3 RFCs are regular IETF contributors. This observation motivated a 
secondary analysis of all the RFCs published in the Independent Stream in 2018. There are 14 
such RFCs: 8507, 8494, 8493, 8492, 8483, 8479, 8433, 8409, 8374, 8369, 8367, 8351, 8328, and 8324. 
(RFCs 8367 and 8369 were published on 1 April 2018.) The majority of the documents were 
published by regular IETF participants, but two of them were not. One describes "The BagIt File 
Packaging Format (V1.0)" [RFC8493], and the other the "Yeti DNS Testbed" [RFC8483]. They 
document a data format and a system developed outside the IETF and illustrate the outreach 
function of the Independent Stream. In both cases, the authors include one experienced IETF 
participant, who presumably helped outsiders navigate the publication process. 


The present document experienced some publication delays due to the Independent Submissions 
Editor. The ISE is a bottleneck and is a volunteer resource. Although the ISE as a lone person 
operating as a volunteer is still roughly adequate resource for the job, the delivery will 
necessarily be best effort with delays caused by spikes in ISE load, work commitments, and other 
life events. These delays may not be fundamentally critical to RFC delivery, but they are capable 
of introducing a significant percentage delay into what might otherwise be a smooth process. 


5. Citation Counts 


In this exploration, we want to examine whether citation counts provide a meaningful 
assessment of the popularity of RFCs. We obtain the citation counts through the Semantic Scholar 
API, using queries of the form: <https://api.semanticscholar.org/v1/paper/10.17487/rfc8446? 
include_unknown_references=true> 


In these queries, the RFC is uniquely identified by its DOI reference, which is composed of the 
RFC Series prefix 10.17487 and the RFC identifier. The queries return a series of properties, 
including a list of citations for the RFC. Based on that list of citations, we compute three numbers: 


e The total number of citations 
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e The number of citations in the year of publication and the year after that 


e For the RFC published in 1998 or 2008 that we use for comparison, the number of citations in 
the years 2018 and 2019. 


All the numbers were retrieved on October 6, 2019. 


5.1. Citation Numbers 


As measured on October 6, 2019, the citation counts for the RFC in our sample set were: 


RFC (2018) Status Total 2018-2019 


8411 Info 1 0 
8456 Info 1 1 
8446 PS 418 204 
8355 Info 3 3 
8441 BS 1 1 
8324 Info (ISE) 0 0 
8377 PS 0 0 
8498 Info 0 0 
8479 Info (ISE) 0 0 
8453 Info 3 3 
8429 BCP 0 0 
8312 Info 25 16 
8492 Info (ISE) 4 4 
8378 Exp 1 1 
8361 PS 0 0 
8472 PS 1 1 
8471 PS 1 1 
8466 PS 0 0 
8362 BS 1 1 
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RFC (2018) Status Total 2018-2019 


8468 Info 1 1 
Table 10 


The results indicate that [RFC8446] is by far the most cited of the 20 RFC in our sample. This is 
not surprising, since TLS is a key Internet Protocol. The TLS 1.3 protocol was also the subject of 
extensive studies by researchers, and thus was mentioned in a number of published papers. 
Surprisingly, the Semantic Scholar mentions a number of citations that predate the publication 
date. These are probably citations of the various draft versions of the protocol. 


The next most cited RFC in the sample is [RFC8312] which describes the Cubic congestion control 
algorithm for TCP. That protocol was also the target of a large number of academic publications. 
The other RFCs in the sample only have a small number of citations. 


There is probably a small bias when measuring citations at a fixed date. An RFC published in 
January 2018 would have more time to accrue citations than one published in December. That 
may be true to some extent, as the second most cited RFC in the set was published in January. 
However, the effect has to be limited. The most cited RFC was published in August, and the 
second most cited was published in 2019. (That RFC got an RFC number in 2018, but publication 
was slowed by long AUTH48 delays.) 


5.2. Comparison to 1998 and 2008 


In order to get a baseline, we can look at the number of references for the RFCs published in 
2008 and 1998. However, we need to take time into account. Documents published a long time 
ago are expected to have accrued more references. We try to address this by looking at three 
counts for each document: the overall number of references over the document's lifetime, the 
number of references obtained in the year following publication, and the number of references 
observed since 2018: 


RFC (2008) Status Total 2008-2009 2018-2019 


5326 Exp 138 14 15 
5348 PS 14 3 0 
5281 Info 69 15 7 
5354 Exp 17 13 0 
S22 PS 19 1 2 
5329 PS 24 6 1 
S21 BS 32 3 2 
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RFC (2008) Status Total 2008-2009 


5236 Info (ISE) 25 5 
5358 BCP 21 2 
5271 Info 7 2 
5195 PS 7 4 
5283 PS 8 1 
5186 Info 14 4 
5142 PS 8 4 
5373 PS 5 2 
5404 PS 1 T 
5172 PS 2 0 
5349 Info 8 0 
5301 PS 5 1 
5174 Info 0 0 
Table 11 


RFC (1998) Status Total 1998-1999 


2289 PS 2 0 
2267 Info 982 5 
2S1 BCP 9 af 
2404 PS 137 6 
2374 PS 42 4 
2449 PS 7 2 
2283 PS 1y 3 
2394 Info 13 2 
2348 DS 5 0 
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2018-2019 
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RFC (1998) Status Total 1998-1999 
2382 Info 17 12 
2297 Info (ISE) 36 lal 
2381 PS 39 12 
2312 Info 14 3 
2387 PS 4 1 
2398 Info 17 0 
2391 PS 31 3 
2431 PS 3 0 
2282 Info 8 0 
2323 Info (ISE) 1 0 
2448 Info (ISE) 0 0 
Table 12 


January 2021 


2018-2019 
0 


0 


We can compare the median number of citations and the numbers of citations for the least and 
most popular quartiles in the three years: 


Huitema 


References Lower 25% Median 
RFC (2018) 0 1 
RFC (2008) 6.5 11 
RFC (2008), until 2009 1 25 
RFC (2008), 2018 and after 0 0 
RFC (1998) 4.75 13.5 
RFC (1998), until 1999 0 2 
RFC (1998), 2018 and after 0 0 
Table 13 
Informational 


Higher 25% 
3 


21.75 


Page 33 


RFC 8963 RFC Evaluation 2018 January 2021 


The total numbers show new documents with fewer citations than the older ones. This can be 
explained to some degree by the passage of time. If we restrict the analysis to the number of 
citations accrued in the year of publishing and the year after that, we still see about the same 
distribution for the three samples. 


We also see that the number of references to RFCs fades over time. Only the most popular of the 
RFC produced in 1998 are still cited in 2019. 


5.3. Citations versus Deployments 


The following table shows side by side the number of citations as measured in Section 5.1 and 
the estimation of deployment as indicated in Section 3. 


RFC (2018) Status Citations Deployment 


8411 Info 1 medium 
8456 Info 1 medium 
8446 PS 418 high 
8355 Info 3 medium 
8441 PS íl high 
8324 Info (ISE) 0 N/A 
8377 PS 0 unknown 
8498 Info 0 unknown 
8479 Info (ISE) 0 one 
8453 Info 3 unknown 
8429 BCP 0 some 
8312 Info 25 high 
8492 Info (ISE) 4 one 
8378 Exp 1 some 
8361 PS 0 one 
8472 PS 1 medium 
8471 PS 1 medium 


Huitema Informational Page 34 


RFC 8963 RFC Evaluation 2018 January 2021 


RFC (2018) Status Citations Deployment 


8466 PS 0 unknown 

8362 PS 1 medium 

8468 Info 1 some 
Table 14 


From looking at these results, it is fairly obvious that citation counts cannot be used as proxies 
for the "value" of an RFC. In our sample, the two RFCs that have high citation counts were both 
widely deployed, and can certainly be described as successful, but we also see many RFCs that 
saw significant deployment without garnering a high level of citations. 


Citation counts are driven by academic interest, but are only loosely correlated with actual 
deployment. We saw that [RFC8446] was widely cited in part because the standardization process 
involved many researchers, and that the high citation count of [RFC8312] is largely due to the 
academic interest in evaluating congestion control protocols. If we look at previous years, the 
most cited RFC in the 2008 sample is [RFC5326], an experimental RFC defining security 
extensions to an experimental delay tolerant transport protocol. This protocol does not carry a 
significant proportion of Internet traffic, but has been the object of a fair number of academic 
studies. 


The citation process tends to privilege the first expression of a concept. We see that with the most 
cited RFC in the 1998 set is [RFC2267], an informational RFC defining Network Ingress Filtering 
that was obsoleted in May 2000 by [RFC2827]. It is still cited frequently in 2018 and 2019, 
regardless of its formal status in the RFC Series. We see the same effect at work with [RFC8441], 
which garners very few citations although it updates [RFC6455] that has a large number of 
citations. The same goes for [RFC8468], which is sparsely cited while the [RFC2330] is widely 
cited. Just counting citations will not indicate whether developers still use an old specification or 
have adopted the revised RFC. 


5.4. Citations versus Web References 


Web references might be another indicator of the popularity of an RFC. In order to evaluate 
these references, we list here the number of results returned by searches on Google and Bing, 
looking for the search term "RFCnnnn'" (e.g., "RFC8411"), and copying the number of results 
returned by the search engines. The table below presents the results of these searches, 
performed on April 4, 2020. 


RFC (2018) Status Citations Google Bing 
8411 Info íl 301 94 


8456 Info 1 266 8456 
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RFC (2018) 
8446 
8355 
8441 
8324 
8377 
8498 
8479 
8453 
8429 
8312 
8492 
8378 
8361 
8472 
8471 
8466 
8362 


8468 
Table 15 


RFC Evaluation 2018 


Status Citations 
PS 418 
Info 3 
PS it 
Info (ISE) 0 
PS 0 
Info 0 
Info (ISE) 0 
Info 3 
BCP 0 
Info 25 
Info (ISE) 4 
Exp 1 
PS 0 
PS 1 
PS 1 
PS 0 
PS íl 
Info 1 


Google 
25900 
521 
2430 
393 
264 
335 
564 
817 
391 
1620 
323 
418 
499 
496 
1510 
766 
67 


453 


Bing 
47800 
114 
59500 
138 
10900 
10100 
11000 
11400 
41600 
2820 
9400 
11600 
92 
169 
11600 
173 
147 


127 


January 2021 


The result counts from Bing are sometimes surprising. Why would RFC 8441 gather 59,500 web 


references? Looking at the results in detail, we find a mix of data. Some of them are logs of 


development projects implementing Web Sockets, which is exactly what we are looking for, but 


others appear spurious. For example, a shop selling rugby jerseys is listed because its phone 


number ends with "8441". Other pages were listed because street numbers or product numbers 
matched the RFC number. The same type of collision may explain the large reference counts on 


Bing for RFCs 8377, 8498, 8479, 8453, 8429, 8378, and 8471. The result counts on Bing do not 
appear to provide a good metric. 


Huitema 


Informational 


Page 36 


RFC 8963 RFC Evaluation 2018 January 2021 


On Google, all RFCs garner at least a 250 references, largely because the whole RFC catalog is 
replicated on a large number of web servers. Deviations from that baseline are largely correlated 
with the number of citations in the Semantic Scholar, with a couple of exception: RFC 8441 and 
RFC 8471 garner more references than the low citation counts would predict. Looking at the 
results, we find many references in development databases explaining how these protocols are 
implemented in various code bases and open source projects. This means that counting Google 
results would give some indication about an RFC's popularity, complementing the citation counts. 


There are some practical problems in using the counts of Google results. Google searches are 
personalized, the results depend on the source of the queries, and the counts may vary as well. 
The search results depend on the search algorithm, and there is no guarantee that counts will not 
change when the algorithm changes. On the other hand, the results do indicate that some of the 
RFCs in our sample are being used by developers or in deployments. 


6. Observations and Next Steps 


The author's goal was to get a personal understanding of the "chain of production" of the RFCs, 
and in particular to look at the various causes of delays in the process. As shown in Section 4, the 
average RFC was produced in 3 years and 4 months, which is similar to what was found in the 
2008 sample, but more than three times larger than the delays for the 1998 sample. 


The working group process appears to be the main source of delays. Efforts to diminish delays 
should probably focus there, instead of on the IETF and IESG reviews or the RFC production. For 
the RFC production phase, most of the variability originates in the AUTH48 process, which is 
influenced by a variety of factors such as number of authors or level of engagement of these 
authors. 


Most of the delay is spent in the working group, but the IETF Datatracker does not hold much 
information about what happens inside the working groups. For example, events like Working 
Group Last Calls were not recorded in the history of the selected drafts available in the 
Datatracker. Such information would have been interesting. Of course, requiring that 
information would create an administrative burden, so there is clearly a trade-off between 
requiring more work from working group chairs and providing better data for process analysis. 
(It appears that this information can be available in the Datatracker for more recent drafts, if the 
WG chairs use the Datatracker properly.) 


The Independent Stream operates as expected. The majority of the authors of the Independent 
Stream RFCs appear to be in IETF insiders, but there is significant amount of engagement by 
outside parties. 


The analysis of citations in Section 5.1 shows that citation numbers are a very poor indication of 
the "value" of an RFC. Citation numbers measure the engagement of academic researchers with 
specific topics, but have little correlation with the level of adoption and deployment of a specific 
RFC. The result counts of Google searches do capture references outside academia, such as logs of 
development projects. This might be informative, but it is not clear that the counts would not 
change over time due to algorithm changes or personalization. 
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This document analyses a small sample of RFCs "in depth". This allowed gathering of detailed 
feedback on the process and the deployments. On the other hand, much of the data on delays is 
available from the IETF Datatracker. It may be worth considering adding an automated reporting 
of delay metrics in the IETF Datatracker. 


This document only considers the RFCs that were published in a given year. This approach can 
be criticized as introducing a form of "survivor bias". There are many drafts proposed to the 
IETF, and only a fraction of them end up being published as RFCs. On one hand, this is expected, 
because part of the process is to triage between ideas that can gather consensus and those that 
don't. On the other hand, we don't know whether that triage is too drastic and has discouraged 
progress on good ideas. 


One way to evaluate the triage process would be to look at publication attempts that were 
abandoned -- for example, drafts that expired without progressing or being replaced. The 
sampling methodology could also be used for that purpose. Pick maybe 20 drafts at random, 
among those abandoned in a target year, and investigate why they were abandoned. Was it 
because better solutions emerged in the working group? Or maybe because the authors 
discovered a flaw in their proposal? Or was it because some factional struggle blocked a good 
idea? Was the idea pursued in a different venue? Hopefully, someone will try this kind of 
investigation. 


7. Security Considerations 


This document does not specify any protocol. 


We might want to analyze whether security issues were discovered after publication of specific 
standards. 


8. IANA Considerations 


This document has no IANA actions. 


Preliminary analysis does not indicate that IANA is causing any particular delay in the RFC 
publication process. 
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